Predicting accuracy of submitted data

ABSTRACT

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for predicting the accuracy of user submissions. One of the methods includes receiving, from a user, an update to an attribute of an entity. If the user is determined to be reliable based on user profile data of the user, the knowledge base is updated with the update to the attribute of the entity.

CROSS-REFERENCE TO RELATED APPLICATION

This is a continuation of U.S. application Ser. No. 16/291,589, filed onMar. 4, 2019, which is a continuation of U.S. application Ser. No.13/906,220, filed on May 30, 2013 (now U.S. Pat. No. 10,223,637), thedisclosures of which are considered part of and are incorporated byreference in the disclosure of this application in their entireties.

BACKGROUND

This specification relates to determining whether a submission of databy a user is accurate.

A search system can provide one or more knowledge panels in response toa received search query. A knowledge panel is a user interface elementthat provides a collection of information or other content related to aparticular entity referenced by the search query. For example, theentity may be a person, place, country, landmark, animal, historicalevent, organization, business, sports team, sporting event, movie, song,album, game, work of art, or any other entity.

In general, a knowledge panel provides a summary of information aboutthe entity. For example, a knowledge panel for a famous singer mayinclude the name of the singer, an image of the singer, a description ofthe singer, one or more facts about the singer, content that identifiessongs and albums recorded by the singer, and/or links to searchesrelated to the singer. Other types of information and content can alsobe presented in the knowledge panel. Information presented in aknowledge panel can include content obtained from multiple disparatesources, e.g., multiple different web pages accessible over theInternet.

A search system can maintain a knowledge base that stores informationabout various entities. The system can assign a unique entity identifierto each entity. The system can also assign one or more text stringaliases to a particular entity. For example, the Statue of Liberty canbe associated with aliases “the Statue of Liberty” and “Lady Liberty.”Aliases need not be unique among entities. For example, “jaguar” can bean alias both for an animal and for a car manufacturer.

The system can also store information about an entity's relationship toother entities. For example, the system can define a “located in:”relationship between two entities to reflect, for example, that theStatue of Liberty is located in New York City. In some implementations,the system stores relationships between entities in a representation ofa graph in which nodes represent distinct entities and links betweennodes represent relationships between the entities. In this example, thesystem could maintain a node representing the Statue of Liberty, a noderepresenting New York City, and a link between the nodes to representthat the Statue of Liberty is located in New York City.

SUMMARY

This specification describes how a system can compute a likelihood thata user will provide accurate updates to a knowledge base based oninformation in the user's profile. In general, the system can train amodel using previous knowledge base submissions by users and use themodel to predict whether a particular user will provide accurateupdates.

In general, one innovative aspect of the subject matter described inthis specification can be embodied in methods that include the actionsof receiving, from a user, an update to an attribute of an entityrelated to a topic; obtaining user profile data of the user; determiningfrom the user profile data that the user is reliable relative to thetopic; and in response to determining that the user is reliable relativeto the topic, updating a knowledge base with the update to the attributeof the entity. Other embodiments of this aspect include correspondingcomputer systems, apparatus, and computer programs recorded on one ormore computer storage devices, each configured to perform the actions ofthe methods. A system of one or more computers can be configured toperform particular operations or actions by virtue of having software,firmware, hardware, or a combination of them installed on the systemthat in operation causes or cause the system to perform the actions. Oneor more computer programs can be configured to perform particularoperations or actions by virtue of including instructions that, whenexecuted by data processing apparatus, cause the apparatus to performthe actions.

The foregoing and other embodiments can each optionally include one ormore of the following features, alone or in combination. Determiningthat the user is reliable relative to the topic comprises computing,using the user profile data as input to a user model, a likelihood thatan update from the user to an entity related to the topic is accurate;and determining that the computed likelihood satisfies a threshold. Theuser model is trained using training examples that represent previouslysubmitted updates to the knowledge base by users and whether thepreviously submitted updates were accurate. Each training exampleincludes information from a user profile of a user that submitted thecorresponding update. The information from the user profile includes oneor more statistics describing the accuracy of knowledge base submissionsby the user or a topic of interest and a level of expertise for thetopic of interest. The information from the user profile includesinformation about subsystems accessed by the user. The update to theattribute of the entity includes an update to a value of an existingattribute of the entity stored in the knowledge base. The update to theattribute of the entity includes a new attribute of the entity that waspreviously not stored in the knowledge base. The threshold is differentfor an existing attribute of the entity than for a new attribute for theentity. The updated entity attribute in the knowledge base is providedin response to search requests by users.

In general, another innovative aspect of the subject matter described inthis specification can be embodied in methods that include the actionsof receiving, from a user, a search request related to a topic;obtaining user profile data of the user; determining from the userprofile data that the user is reliable relative to the topic; inresponse to determining that the user is reliable relative to the topic,providing to the user a request for an update to an attribute of anentity related to the topic; receiving, from the user, an update to theattribute of the entity related to the topic; and updating a knowledgebase with the update to the attribute of the entity. Other embodimentsof this aspect include corresponding computer systems, apparatus, andcomputer programs recorded on one or more computer storage devices, eachconfigured to perform the actions of the methods.

The foregoing and other embodiments can each optionally include one ormore of the following features, alone or in combination. Determiningthat the user is reliable relative to the topic comprises computing,using the user profile data as input to a user model, a likelihood thatan update from the user to an entity related to the topic is accurate;and determining that the computed likelihood satisfies a threshold. Theuser model is trained using training examples that represent previouslysubmitted updates to the knowledge base by users and whether thepreviously submitted updates were accurate. Each training exampleincludes information from a user profile of a user that submitted thecorresponding update. The information from the user profile includes oneor more statistics describing the accuracy of knowledge base submissionsby the user or a topic of interest and a level of expertise for thetopic of interest. The information from the user profile includesinformation about subsystems accessed by the user. Providing to the usera request for an update to an attribute of an entity related to thetopic comprises providing a knowledge panel that presents one or moreitems of information about the entity and requests the update to theattribute of the entity. Receiving, from a user, an update to anattribute of an entity related to a topic comprises receiving the updateto the attribute of the entity through a user interface control of theknowledge panel.

The subject matter described in this specification can be implemented inparticular embodiments so as to realize one or more of the followingadvantages. A search system can automatically determine whether asubmission from a user is likely to be accurate based on the accuracy ofprevious submissions received from the user or other indications of usertrustworthiness. This can reduce the amount of erroneous or spam inputsto the knowledge base. A search system is more likely to receiveaccurate data updates for a particular topic by asking users who areinterested in the particular topic to provide updates on the topic. Thiscan reduce the likelihood that a user will be annoyed by being asked toprovide an update and can increase the likelihood of receiving aresponse from a user.

The details of one or more embodiments of the subject matter of thisspecification are set forth in the accompanying drawings and thedescription below. Other features, aspects, and advantages of thesubject matter will become apparent from the description, the drawings,and the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an example search results page that includes aknowledge panel.

FIG. 2 is a diagram of an example system.

FIG. 3 is a flow chart of an example process for training a user model.

FIG. 4 is a flow chart of an example process for computing thelikelihood that a user will provide an accurate update.

FIG. 5 is a flow chart of an example process for asking particular usersto update knowledge base information.

Like reference numbers and designations in the various drawings indicatelike elements.

DETAILED DESCRIPTION

Some search systems allow users to update data stored in a knowledgebase. In general, an update to the knowledge base updates an [attribute,value] pair associated with an entity. For example, a user can submit anupdate to the knowledge base for an existing attribute “date of birth,”where the updated value is “Feb. 12, 1809.” A user may also submit anupdate to the knowledge base for a new attribute associated with aperson entity, e.g., “Favorite food,” and a corresponding new value,e.g., “pizza,” providing both the new attribute and the new value.

However, in some cases it may be difficult for the search system todetermine whether data entered by a user is correct, as the user mayintentionally or unintentionally enter incorrect data.

A search system can create a user model to determine whether aparticular user is likely to provide accurate updates to the knowledgebase. The search system can use machine learning to generate the usermodel based on the accuracy of updates to the knowledge base previouslyentered by the user and profile information associated with the user.Once the user model is developed, the search system can use the usermodel to predict the accuracy of knowledge base updates submitted by theuser, determine whether to update the knowledge base with the submitteddata, and determine whether or not to ask a the user for specific dataor verification of data in the knowledge base.

FIG. 1 illustrates an example search results page 100 that includes aknowledge panel 130. A user can submit the query 102 to a search systemthrough a graphical user interface of a software application, e.g., aweb browser, or through a user interface of some other softwareapplication installed on a user device, e.g., a spoken query issuedthrough a speech recognition application installed on a mobile userdevice. In response to receiving the query 102, the search system canprovide a search results page 100 in a form that can be presented on theuser device. For example, the search results page 100 can be provided asa markup language document, e.g., a HyperText Markup Language document,and the user device can render the document, e.g., using a web browser,in order to present the search results page 100 on a display of the userdevice.

The search results page 100 includes three search results 122 a-c thatthe search system has obtained in response to the query 102. Each of thesearch results 122 a-c includes a title, a display link, and a textsnippet. Each of the search results 122 a-c is also linked to arespective resource, e.g., a web page at a location indicated by thedisplay link. User selection of a search result will cause theapplication to navigate to the linked resource. The search results page100 also includes an indicator 110 that the user is currently logged in.

The search results page 100 also includes a knowledge panel 130corresponding to an entity with an alias corresponding to the searchquery 102. In this example, the entity is Abraham Lincoln.

The knowledge panel 130 includes various items of information aboutAbraham Lincoln. The knowledge panel 130 includes an entity name 132, apicture of the entity 133, items of information 134, including anoccupation, a date of birth, a date of death, and a spouse's name.

The search system can provide the knowledge panel 130 as an interfacefor the user to update one or more items of information maintained bythe search system in the knowledge base. For example, the search systemcan invite the user to correct a specific one of the items ofinformation 134, or the search system can, upon user selection of any ofthe items of information 134, provide an editable text-input field 136for editing the item of information. For example, upon user selection ofthe “Spouse” field, the search system can provide editable text-inputfield 136 through which the user can edit that particular item ofinformation.

After making changes to the information in the knowledge panel 130, theuser can submit the information, e.g., by selecting a “Submit” userinterface control 138. The system can then evaluate the submittedinformation based on one or more criteria, e.g., the user's reliabilityor data submitted by other users. If the system determines that theupdate is likely to be accurate, the system can update the knowledgebase with the submitted information. In this way, the system can use theknowledge panel 130 as an efficient way to ask for updates toinformation maintained by the search system from one place and in-line,e.g., without having to navigate away from the search results page 100.

FIG. 2 is a diagram of an example system 200. In general, the systemincludes a user device 210 coupled to a search system 230 over a network220. The search system 230 is an example of an information retrievalsystem in which the systems, components, and techniques described belowcan be implemented.

In operation, the user device 210 transmits a query 212 to the searchsystem 230, e.g., over the network 220. The query 212 includes one ormore terms and can include other information, for example, a locationand a type of the user device 210. The search system 230 generates aresponse, generally in the form of a search results page 216. The searchresults page 216 can include search results 213 that the search system230 has identified as being responsive to the query 212.

If the search system 230 determines that the user is likely to know andprovide accurate information about a particular entity, e.g., an entityrelevant to a user's field of expertise, the search system 230 canprovide a data request 214 that requests an update to a particular itemof information about the entity in the knowledge base 262. In someimplementations, the data request 214 can be included in a knowledgepanel for the entity, which can be used as an interface for the user toupdate the requested items of information. The search system 230transmits the search results page 216 over the network 220 back to theuser device 210 for presentation to a user.

The search system 230 can receive updated information 218 that is eitherinitiated by the user or initiated by a data request 214. The updatedinformation can be received, for example, through a knowledge panelprovided on the search results page 216. The search system 230 can thenuse the updated information 218 to update the knowledge base 262.

The user device 210 can be any appropriate type of computing device,e.g., mobile phone, tablet computer, notebook computer, music player,e-book reader, laptop or desktop computer, PDA (personal digitalassistant), smart phone, a server, or other stationary or portabledevice, that includes one or more processors 208 for executing programinstructions and memory 206, e.g., random access memory (RAM). The userdevice 210 can include non-volatile computer readable media that storesoftware applications, e.g., a browser or layout engine, an inputdevice, e.g., a keyboard or mouse, a communication interface, and adisplay device.

The network 220 can be, for example, a wireless cellular network, awireless local area network (WLAN) or Wi-Fi network, a mobile telephonenetwork or other telecommunications network, a wired Ethernet network, aprivate network such as an intranet, a public network such as theInternet, or any appropriate combination of such networks.

The search system 230 can be implemented as computer programs installedon one or more computers in one or more locations that are coupled toeach through a network. The search system 230 includes a search systemfront end 240, a search engine 250, a data request module 260, and amachine learning module 270. The computing device or devices thatimplement the search system front end 240, the search engine 250, thedata request module 260, and the machine learning module 270 may includesimilar components.

The search system 230 includes a user database 272 that storesinformation about users who access the search system 230. For example,the user database 272 may include a user profile for each of the userswho access the search system. For users who are registered users of thesearch system 230, a user profile can include previous submissions bythe user to the knowledge base and whether such submissions wereaccurate or not. A user profile for registered or unregistered users mayinclude user interactions with subsystems of the search system, e.g., aweb search system, an image search system, a map system, an emailsystem, a social network system, a blogging system, a shopping system,just to name a few, topics of interest, and an indication of a level ofexpertise of the user for each of the topics of interest, e.g., noviceor expert. The topics of interest and levels of expertise may includeuser-provided data or system-generated data based on a user'sinteraction with the search system. For example, the search system maydetermine that a specific user is interested in French restaurants basedon a search history of the specific user or search results selected bythe specific user. The search system may then add “restaurants” to theuser's topics of interest.

In some implementations, users are distinguished by the IP addresses ofthe user devices used in performing the activities. In someimplementations, activities are recorded by the interactive systeminvolved in the activity. In some implementations, activity informationis also, or alternatively, collected with the consent of the user by anapplication, e.g, a web browser toolbar, running on the user's device.

Where personal information about users may be collected or used, usersmay be given an opportunity to control whether the personal informationabout the users is collected. In addition, certain data may be treatedin one or more ways before it is stored or used, so that personallyidentifiable information is removed.

In general, the search system front end 240 receives the query 212 fromthe user device 210 and routes the query 212 to the search engine 250and the data request module 260. The search system front end 240 alsoprovides the resulting search results page 216 that includes the searchresults 213 and the knowledge panel 214 to the user device 210. In doingso, the search system front end 240 acts as a gateway, or interface,between user devices and the search system 230.

The search engine 250 receives the query 212 and generates searchresults 213 that are responsive to the query. The search engine 250 willgenerally include an indexing engine for indexing resources in acollection of resources. For example, the search engine 250 can indexweb pages found in a collection of web pages, e.g., web pages on theInternet. A collection of resources indexed by the indexing engine may,but need not, be stored within search system 230, e.g., in indexdatabase 252. The search engine 250 can rank the search results 213using conventional methods and route the ranked search results 213 backto search system front end 240 for inclusion in the search results page216.

The data request module 260 receives the query 212 and determineswhether the search system 230 should provide a knowledge panel in aresponse to the query as well as whether to ask the user through a datarequest 214 to update information about a particular entity. In someimplementations, the data request 214 is presented through a knowledgepanel on the search results page 216.

The data request module 260 can determine whether the search system 230should provide a knowledge panel using a data structure of the knowledgebase 262 that maps an alias to one or more entities, e.g. an entityalias index. For example, the alias “Bush” can be mapped to a set ofentities having that alias, e.g., the entity “George W. Bush,” theentity “George H. W. Bush,” the entity for the rock band “Bush,” and theentity for a category of plants having that alias. The entity aliasindex may also include a score for each entity that represents alikelihood that the alias refers to each particular entity. The datarequest module 260 can use some or all of the query 212 as input to theentity alias index. The data request module 260 can use a returnedentity for the query to present an entity in a knowledge panel. The datarequest module 260 can also use a returned entity to identify a topic ofthe query, which can be used to compute a likelihood that the user willprovide accurate updates for the topic.

The data request module 260 evaluates the accuracy of the updatedinformation 218 to determine whether to update the knowledge base 262with the updated information 218. The data request module 260 candetermine both whether to provide a data request 214 as well as theaccuracy of the updated information 218 by using a user model 217generated by the machine learning module 270.

The machine learning module 270 receives the user profile data from userdatabase 272, and generates a user model that predicts the likelihoodthat a particular user will submit accurate information for theknowledge base. The user model can be trained using one or more items ofinformation in the user profiles, e.g., previous knowledge basesubmissions on various topics, the accuracy of such submissions, topicsof interest of the users, and a level of expertise of the users for eachtopic of interest.

FIG. 3 is a flow chart of an example process for training a user model.The system receives previous knowledge base submissions by users anduser profile data of the submitting users. The system then trains a usermodel that can be used to compute a likelihood that a user havingparticular user profile data will provide an accurate update for atopic. The process can be implemented by one or more computer programsinstalled on one or more computers. The process will be described asbeing performed by a system of one or more computers, e.g. the machinelearning module 270 of FIG. 2 .

The system obtains previous knowledge base submissions (310). The systemcan train the user model using training data that includes previousknowledge base submissions on various topics along with information fromprofiles of the users who provided the submissions. The training dataincludes training examples that each represents a previous usersubmission and one or more features of that particular submission. Thefeatures can include a topic of the previous user submission and one ormore items of information from the user's profile, e.g., a measure ofthe accuracy of the user's other knowledge base submissions, topics ofinterest in the user's profile, a level of expertise for each topic ofinterest, and other subsystems used by the user, for example.

Each training example can be labeled to indicate whether the previoussubmission by a user was accurate, e.g., with a score ranging from 0 to1 or with a binary classification as “good”/“bad,” or“reliable”/“unreliable.” In some implementations, the training data ishand-labeled by administrators of the knowledge base.

The system obtains statistics of a previous knowledge base submission(320). One example feature of the training examples is a measure of theaccuracy of previous knowledge base submissions of the user. Forexample, the system can select the previous knowledge base submissionsassociated with the user's profile and determine the accuracy of thosesubmissions, e.g., based on the submissions being added to the knowledgebase or updates being later changed back, e.g. by knowledge basecurators, to a previous version. The accuracy of a particular submissioncan be determined according to whether the submission was added to theknowledge base, for example, by considering a revision history of theknowledge base after the user's submission. The accuracy of a particularsubmission can also be determined by verification by other knowledgebase users, by an expert, or by an administrator of the search system.

The system can compute statistics to indicate the accuracy of theprevious knowledge base submissions that the user has made, for example,a ratio of correct to incorrect submissions. For example, the system canconsider a first user with a high ratio of correct to incorrectsubmissions to be more reliable than a second user with a lower ratio ofcorrect to incorrect submissions.

Other values may be used to represent the accuracy of previoussubmissions that the user has made to the knowledge base.

The system obtains topics of interest and levels of expertise (330).Another example feature for the training examples includes a level ofexpertise for each of the topics of interest in the user's profile. Insome implementations, the system generates the levels of expertiseautomatically. For example, the system can determine a level ofexpertise based on input received from a user and based on the types ofdocuments that the user subsequently accesses. For example, the searchsystem can determine that a user who views highly technical documentsmay be an expert in a particular field, e.g., medicine or technology.Conversely, the search system can determine that a user who views onlygeneral documents associated with the same field is a novice.

The search system can use any appropriate algorithm to determine a levelof expertise for a specific user with respect to a specific topic ofinterest. For example, the search system may use machine learning tocreate an expertise model to determine a level of expertise for topicsassociated with profiles of users who access the knowledge base. Theexpertise model can be trained by using the measure of languagesophistication on resources accessed by users as input in order toclassify resources as those that would be visited by experts or noviceson a particular topic. The system can then use resources visited by auser to determine whether the user is an expert or a novice for thetopic.

The system obtains information about other subsystems accessed by theuser (340). Another example feature for the training examples includesinformation about other subsystems of the search system accessed by auser. In general, a higher number of subsystems accessed by the sameuser is a signal of legitimacy for the associated user. In contrast, auser who has accessed only one subsystem is more likely to be suspect.Thus, a user profile that is associated with the search engine and asocial networking website will generally be more likely to have a highpredicted accuracy than a user profile that is only associated with thesearch engine, assuming all other scoring factors are the same.

The system trains the user model (350). The machine learning module 270uses the labeled training examples to train the user model. The modulecan be implemented with any appropriate supervised learning algorithmthat uses labeled training data, e.g. a support vector machine, logisticregression, or nearest-neighbor classifiers.

In some implementations, the machine learning module performs activelearning and updates the user model as the knowledge base receivesadditional data from users. For example, the machine learning moduleupdates the user model or creates a new user model according to aschedule, e.g., monthly or yearly, or at another predetermined time,e.g., one specified by an administrator.

FIG. 4 is a flow chart of an example process for computing thelikelihood that a user will provide an accurate update. In general, thesystem receives a data update from a user on a particular topic. Thesystem can then determine a likelihood that the user will provideaccurate data on the particular topic using information in the user'sprofile. The process can be implemented by one or more computer programsinstalled on one or more computers. The process will be described asbeing performed by a system of one or more computers, e.g. the datarequest module 260 of FIG. 2 .

The system receives an update from a user for a topic (410). Forexample, the system can receive an update from a user through aknowledge panel provided as part of a search results page, asillustrated in FIG. 1 . The system can determine the topic, for example,by determining one or more entities for which a query submitted by theuser is an alias. The system can also receive an update from a user whois browsing and submitting updates to a knowledge base through a directinterface to the knowledge base, in which case the topic can bedetermined from an entity associated with the update.

The system obtains user profile data of the user (420).

The system determines that the user is reliable relative to the topic(430). A user can be considered reliable relative to a topic if thesystem determines that the user is likely to provide updates to theknowledge base that are accurate. The system can use the obtained userprofile data of the user and the topic of the update as input to a usermodel to determine the likelihood that an update from the user on thetopic is accurate. Generally, if the determined likelihood satisfies athreshold, the system can determine that the user is reliable relativeto the topic. The system can then update the knowledge base accordinglywithout further intervention or inspection by knowledge baseadministrators.

For example, the system can compute features from the user's profiledata and use the features as input to the user model, including theuser's previous knowledge base submissions, topics of interest, etc., asdescribed above. The system can then use the features as input to theuser model to compute a likelihood that the user's update for the topicis accurate.

Some users may not have any information associated with their profiles.Thus, in some implementations, if the system determines that there is noinformation available about the user other than the current submission,the system assigns the user a default likelihood of providing anaccurate update for the topic.

Alternatively, if the system determines that the likelihood does notsatisfy the threshold, the system can seek to verify the submissionusing input from one or more other users before updating the knowledgebase. For example, the system can wait for additional submissions byother users and compute an aggregate likelihood that a particular updateto an attribute is reliable. Once a cumulative likelihood of thesubmissions satisfies a threshold, the system can then determine thatthe knowledge base should be updated with a value provided by the usersubmissions.

If the system receives conflicting updates from two or more differentusers, the system can weight each of the responses to determine aresponse that has the highest probability of being accurate. Forexample, when the system receives an update to a phone number of arestaurant from five different users, the system can determine weightsfor the responses based on the computed likelihood associated with eachof the users. Thus, updates from users with a higher likelihood ofaccuracy can outweigh updates from users with a lower likelihood ofaccuracy.

In some implementations, if the search system receives a submission froma user who has a low computed likelihood of providing a reliable update,the search system discards the submission and does not update theknowledge base. The search system may also maintain records of suchlow-likelihood submissions for aggregation with previous and futuresubmissions by other users.

The system can also use different thresholds for updates to existingattributes and new attributes. For example, if the user submits a newattribute, the system can require a higher likelihood that the user willprovide accurate updates for the topic than it would if the attributewere an existing attribute for the entity.

The system updates a knowledge base with the received data update (440).After determining that the knowledge base should be updated, the systemcan change the value of the attribute as provided by the user.Generally, updating the attribute requires no confirmation by knowledgebase administrators and will cause other users that subsequently accessknowledge base information, e.g. by information presented in a knowledgepanel, to be provided with the updated information. FIG. 5 is a flowchart of an example process for asking particular users to updateknowledge base information. In general, the system receives a searchrequest from a user and determines whether to ask the user to provide anupdate to an attribute of an entity in a knowledge base. The process canbe implemented by one or more computer programs installed on one or morecomputers. The process will be described as being performed by a systemof one or more computers, e.g., the search system 230 of FIG. 2 .

The system receives a search request from a user on a particular topic(510). For example, the system can receive a search query from a userwho is logged into the system. The system can then determine a topicfrom the search query, for example, by determining an entity for whichthe search query is an alias. The system can also receive other types ofsearch requests and determine topics from the other types of searchrequests. For example, the system can receive, from a user, a requestfor news stories, map data, social networking data, or other requestsfor other types of data from one or more subsystems of the system.

The system obtains user profile data of the user (520). The systemdetermines that the user is reliable relative to the topic (530). Thesystem can, for example, use information in the user profile data tocompute features that can be used as input to the user model, asdescribed in more detail above with reference to FIG. 3 . The system canuse the user model to compute a likelihood that the user will provideaccurate data for the particular topic.

The system can also use the user model to determine which users toprovide questions to and when to provide the questions. For example, thesystem can identify multiple users whose search request is relevant to aparticular topic or whose recent search history is relevant to aparticular topic. The system can then rank the users according to theirrespective predicted likelihoods of providing accurate updates toentities related to the particular topic. The system can then choose oneor more highest-ranking users to ask for updates.

The system may also consider a time of day of the received request. Forexample, the system may ask users for updates only during each user'snon-working hours, according to the user's local time. Thus, the systemcan highly rank those users whose request was received duringnon-working hours for the geographic region from which the request wasreceived.

The system provides a request for an update to the user (540). In someimplementations, the system provides a knowledge panel, e.g., asillustrated in FIG. 1 , that asks a user if an element of informationabout a particular entity is correct or incorrect, or invites the userto provide such information in the first instance.

The system receives an update from the user on the topic (550). Forexample, the system can receive an update submitted by the user througha knowledge panel interface.

The system updates a knowledge base with the received update (560).Because the system previously evaluated the likelihood that the userwould provide accurate data for the topic, the system need not againevaluate information in the user's profile to determine a likelihoodthat the update is accurate. However, the system may still compare theupdated data to other sources of data, e.g., updates provided by one ormore other users as described above with reference to FIG. 3 .

Embodiments of the subject matter and the functional operationsdescribed in this specification can be implemented in digital electroniccircuitry, in tangibly-embodied computer software or firmware, incomputer hardware, including the structures disclosed in thisspecification and their structural equivalents, or in combinations ofone or more of them. Embodiments of the subject matter described in thisspecification can be implemented as one or more computer programs, i.e.,one or more modules of computer program instructions encoded on atangible non-transitory program carrier for execution by, or to controlthe operation of, data processing apparatus. Alternatively or inaddition, the program instructions can be encoded on anartificially-generated propagated signal, e.g., a machine-generatedelectrical, optical, or electromagnetic signal, that is generated toencode information for transmission to suitable receiver apparatus forexecution by a data processing apparatus. The computer storage mediumcan be a machine-readable storage device, a machine-readable storagesubstrate, a random or serial access memory device, or a combination ofone or more of them.

The term “data processing apparatus” refers to data processing hardwareand encompasses all kinds of apparatus, devices, and machines forprocessing data, including by way of example a programmable processor, acomputer, or multiple processors or computers. The apparatus can also beor further include special purpose logic circuitry, e.g., an FPGA (fieldprogrammable gate array) or an ASIC (application-specific integratedcircuit). The apparatus can optionally include, in addition to hardware,code that creates an execution environment for computer programs, e.g.,code that constitutes processor firmware, a protocol stack, a databasemanagement system, an operating system, or a combination of one or moreof them.

A computer program, which may also be referred to or described as aprogram, software, a software application, a module, a software module,a script, or code, can be written in any form of programming language,including compiled or interpreted languages, or declarative orprocedural languages, and it can be deployed in any form, including as astand-alone program or as a module, component, subroutine, or other unitsuitable for use in a computing environment. A computer program may, butneed not, correspond to a file in a file system. A program can be storedin a portion of a file that holds other programs or data, e.g., one ormore scripts stored in a markup language document, in a single filededicated to the program in question, or in multiple coordinated files,e.g., files that store one or more modules, sub-programs, or portions ofcode. A computer program can be deployed to be executed on one computeror on multiple computers that are located at one site or distributedacross multiple sites and interconnected by a communication network.

The processes and logic flows described in this specification can beperformed by one or more programmable computers executing one or morecomputer programs to perform functions by operating on input data andgenerating output. The processes and logic flows can also be performedby, and apparatus can also be implemented as, special purpose logiccircuitry, e.g., an FPGA (field programmable gate array) or an ASIC(application-specific integrated circuit).

Computers suitable for the execution of a computer program include, byway of example, general or special purpose microprocessors or both, orany other kind of central processing unit. Generally, a centralprocessing unit will receive instructions and data from a read-onlymemory or a random access memory or both. The essential elements of acomputer are a central processing unit for performing or executinginstructions and one or more memory devices for storing instructions anddata. Generally, a computer will also include, or be operatively coupledto receive data from or transfer data to, or both, one or more massstorage devices for storing data, e.g., magnetic, magneto-optical disks,or optical disks. However, a computer need not have such devices.Moreover, a computer can be embedded in another device, e.g., a mobiletelephone, a personal digital assistant (PDA), a mobile audio or videoplayer, a game console, a Global Positioning System (GPS) receiver, or aportable storage device, e.g., a universal serial bus (USB) flash drive,to name just a few.

Computer-readable media suitable for storing computer programinstructions and data include all forms of non-volatile memory, mediaand memory devices, including by way of example semiconductor memorydevices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks,e.g., internal hard disks or removable disks; magneto-optical disks; andCD-ROM and DVD-ROM disks. The processor and the memory can besupplemented by, or incorporated in, special purpose logic circuitry.

To provide for interaction with a user, embodiments of the subjectmatter described in this specification can be implemented on a computerhaving a display device, e.g., a CRT (cathode ray tube) or LCD (liquidcrystal display) monitor, for displaying information to the user and akeyboard and a pointing device, e.g., a mouse or a trackball, by whichthe user can provide input to the computer. Other kinds of devices canbe used to provide for interaction with a user as well; for example,feedback provided to the user can be any form of sensory feedback, e.g.,visual feedback, auditory feedback, or tactile feedback; and input fromthe user can be received in any form, including acoustic, speech, ortactile input. In addition, a computer can interact with a user bysending documents to and receiving documents from a device that is usedby the user; for example, by sending web pages to a web browser on auser's device in response to requests received from the web browser.

Embodiments of the subject matter described in this specification can beimplemented in a computing system that includes a back-end component,e.g., as a data server, or that includes a middleware component, e.g.,an application server, or that includes a front-end component, e.g., aclient computer having a graphical user interface or a Web browserthrough which a user can interact with an implementation of the subjectmatter described in this specification, or any combination of one ormore such back-end, middleware, or front-end components. The componentsof the system can be interconnected by any form or medium of digitaldata communication, e.g., a communication network. Examples ofcommunication networks include a local area network (LAN) and a widearea network (WAN), e.g., the Internet.

The computing system can include clients and servers. A client andserver are generally remote from each other and typically interactthrough a communication network. The relationship of client and serverarises by virtue of computer programs running on the respectivecomputers and having a client-server relationship to each other. In someembodiments, a server transmits data, e.g., an HTML page, to a userdevice, e.g., for purposes of displaying data to and receiving userinput from a user interacting with the user device, which acts as aclient. Data generated at the user device, e.g., a result of the userinteraction, can be received from the user device at the server.

While this specification contains many specific implementation details,these should not be construed as limitations on the scope of anyinvention or on the scope of what may be claimed, but rather asdescriptions of features that may be specific to particular embodimentsof particular inventions. Certain features that are described in thisspecification in the context of separate embodiments can also beimplemented in combination in a single embodiment. Conversely, variousfeatures that are described in the context of a single embodiment canalso be implemented in multiple embodiments separately or in anysuitable subcombination. Moreover, although features may be describedabove as acting in certain combinations and even initially claimed assuch, one or more features from a claimed combination can in some casesbe excised from the combination, and the claimed combination may bedirected to a subcombination or variation of a subcombination.

Similarly, while operations are depicted in the drawings in a particularorder, this should not be understood as requiring that such operationsbe performed in the particular order shown or in sequential order, orthat all illustrated operations be performed, to achieve desirableresults. In certain circumstances, multitasking and parallel processingmay be advantageous. Moreover, the separation of various system modulesand components in the embodiments described above should not beunderstood as requiring such separation in all embodiments, and itshould be understood that the described program components and systemscan generally be integrated together in a single software product orpackaged into multiple software products.

Particular embodiments of the subject matter have been described. Otherembodiments are within the scope of the following claims. For example,the actions recited in the claims can be performed in a different orderand still achieve desirable results. As one example, the processesdepicted in the accompanying figures do not necessarily require theparticular order shown, or sequential order, to achieve desirableresults. In some cases, multitasking and parallel processing may beadvantageous.

1. A system comprising: one or more computers and one or more storagedevices storing instructions that are operable, when executed by the oneor more computers, to cause the one or more computers to performoperations comprising: maintaining, by the system, a knowledge baseaccessible by multiple users, wherein the knowledge base comprises oneor more attribute-value pairs for a plurality of entities; obtaining,from a user having user profile data stored in the system, a value foran attribute of an entity of the plurality of entities, wherein the userprofile data is not stored in the knowledge base; obtaining the userprofile data for the user, the user profile data including informationrepresenting other subsystems of the system accessed by the user;computing a likelihood that the value is accurate using the informationrepresenting the other subsystems; and updating the knowledge base withthe value in response to determining that the likelihood satisfies athreshold.
 2. The system of claim 1, wherein the likelihood that thevalue is accurate is output from a user reliability model, in responseto an input of the information representing the other subsystems.
 3. Thesystem of claim 2, wherein the user reliability model is configured toconsider users who access more subsystems of the system to be morereliable than users who access fewer subsystems of the system.
 4. Thesystem of claim 2, wherein the user reliability model is trained usingtraining examples comprising respective subsystems of the systemaccessed by each user of a plurality of users and a measure of accuracyof previously submitted updates to the knowledge base by the each user.5. The system of claim 1, wherein the other subsystems include an imagesearch system, a map system, an email system, a social network system, ablogging system, or a shopping system.
 6. The system of claim 1, whereinthe other subsystems of the system do not include a search engine or theknowledge base.
 7. The system of claim 1, wherein the updating of theknowledge base with the value comprises automatically updating theknowledge base without intervention or inspection by a knowledge baseadministrator.
 8. A method performed by a search system comprising oneor more computers, the method comprising: maintaining, by the searchsystem, a knowledge base accessible by multiple users, wherein theknowledge base comprises information about entities, the informationabout each entity of the entities being represented as one or moreattribute-value pairs; receiving, by the search system from a userhaving user profile data stored in the search system, an updated valueof an attribute of an entity in the knowledge base, wherein the userprofile data is not stored in the knowledge base; obtaining the userprofile data for the user, the user profile data including informationrepresenting other subsystems of the search system accessed by the user;computing a likelihood that the updated value is accurate using theinformation representing the other subsystems of the search system; andupdating the knowledge base with the updated value in response todetermining that the likelihood satisfies a threshold.
 9. The method ofclaim 8, wherein the likelihood that the updated value is accurate isoutput from a user reliability model, in response to an input of theinformation representing the other subsystems.
 10. The method of claim9, wherein the user reliability model is configured to consider userswho access more subsystems of the search system to be more reliable thanusers who access fewer subsystems of the search system.
 11. The methodof claim 9, wherein the user reliability model is trained using trainingexamples comprising respective subsystems of the search system accessedby each user of a plurality of users and a measure of accuracy ofpreviously submitted updates to the knowledge base by the each user. 12.The method of claim 8, wherein the other subsystems include an imagesearch system, a map system, an email system, a social network system, ablogging system, or a shopping system.
 13. The method of claim 8,wherein the other subsystems of the search system do not include asearch engine or the knowledge base.
 14. The method of claim 8, whereinthe updating of the knowledge base with the updated value comprisesautomatically updating the knowledge base without intervention orinspection by a knowledge base administrator.
 15. The method of claim 8,wherein the receiving of the updated value in the knowledge basecomprises receiving the updated value by a web search engine of thesearch system.
 16. The method of claim 8, wherein the receiving of theupdated value comprises receiving the updated value through a knowledgepanel user interface provided by the web search engine in response tothe user submitting a search query.
 17. The method of claim 8, whereinthe knowledge panel user interface presents one or more items ofinformation about the entity in the knowledge base.
 18. One or morenon-transitory computer storage media encoded with computer programinstructions that when executed by one or more computers of a searchsystem cause the one or more computers to perform operations comprising:maintaining, by the search system, a knowledge base accessible bymultiple users, wherein the knowledge base comprises information aboutentities, the information about each entity of the entities beingrepresented as one or more attribute-value pairs; receiving, by thesearch system from a user having user profile data stored in the searchsystem, a value of an attribute of an entity in the knowledge base,wherein the user profile data is not stored in the knowledge base;obtaining the user profile data for the user, the user profile dataincluding information representing other subsystems of the search systemaccessed by the user, wherein the other subsystems include an imagesearch system, a map system, an email system, a social network system, ablogging system, or a shopping system; computing a likelihood that thevalue is accurate using the information representing the othersubsystems; and updating the knowledge base with the updated valuereceived from the user in response to determining that the likelihoodsatisfies a threshold.
 19. The one or more non-transitory computerstorage media of claim 18, wherein the likelihood that the value isaccurate is output from a trained user reliability model, in response toan input of the information representing the other subsystems.
 20. Theone or more non-transitory computer storage media of claim 19, whereinthe trained user reliability model is configured to consider users whoaccess more subsystems of the search system to be more reliable thanusers who access fewer subsystems of the search system.